Conversation
This was referenced Sep 9, 2026
Introduce API registration, request context, version validation, and asynchronous error mapping. Fix request buffer ownership and response serialization cleanup while preserving the existing ApiVersions entry point. Validated with mvn -o -pl fluss-rpc,fluss-kafka verify. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 555/555 AI-Contributed/UT: 525/525
Catch failures while mapping asynchronous handler results so the dispatcher future completes exceptionally instead of remaining pending. Cover handler futures that fail before and after dispatch registers its callback. Validated with Java 11: Maven reactor build and 17 targeted Kafka tests, including Checkstyle, Spotless, and license checks. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 27/27 AI-Contributed/UT: 28/28
Route requests through the dispatcher and advertise only implemented APIs. Return version-aware errors for unsupported APIs and invalid requests. Validated with mvn -o -pl fluss-kafka verify (23 unit tests and 1 IT). Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 313/313 AI-Contributed/UT: 137/137
Extract topic identity and the raw/string table mapping contract before Metadata. Validate table kinds, field projections and metadata columns independently of request handling and record decoding. Validation: Java 11, mvn -o -pl fluss-kafka clean verify (47 unit tests and 2 integration tests); Checkstyle, Spotless and RAT passed. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 769/769 AI-Contributed/UT: 480/480
Resolve database.table topic names directly to Fluss table paths and return qualified names for metadata. Validate name boundaries and cover same-named tables in different databases through native metadata round trips. Read projected field names directly from DataField to avoid rebuilding the complete field-name list for every projected column. Validated on Java 11: 67 Kafka unit tests and 2 integration tests passed, along with Checkstyle, Spotless, RAT, and git diff --check. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 52/52 AI-Contributed/UT: 167/167
Integrate the DDL mapping prerequisite and use its resolver for Metadata discovery. Omit unsupported tables from all-topic queries and return per-topic mapping errors for named queries. Preserve metadata for compatible tables in mixed requests. Validation: Java 11, mvn -o -pl fluss-kafka clean verify (62 unit tests and 2 integration tests); Checkstyle, Spotless and RAT passed. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 742/742 AI-Contributed/UT: 571/571
Route Metadata queries through the database.table mapping from PR03 and enumerate compatible topics across databases. Preserve brokers and valid topics when a requested database is absent or disappears during discovery; propagate other gateway failures. Cover qualified names, same-named tables across databases, synchronous and asynchronous lookup failures, and Metadata v0-v11 over a real listener. Validated on Java 11: 90 unit tests and 4 integration tests passed, along with Checkstyle, Spotless, RAT, and git diff --check. Five additional interleaved asynchronous review probes passed. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 129/129 AI-Contributed/UT: 514/514
gyang94
force-pushed
the
codex/kafka-metadata-api
branch
from
September 17, 2026 06:40
050b260 to
988918e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #4266. Part of #4185.
Expose pre-created Fluss tables through Kafka Metadata using the shared
database.tableDDL mapping contract. Queries discover compatible tables across databases and preserve brokers and valid topics when another requested table or database is missing.This is PR04, depending on PR03 DDL table mapping (#4277 / #4275), which builds on PR02 ApiVersions (#4260 / #4265) and PR01 (#4259 / #4264).
Brief change log
table_jsonfield. Reject bare or malformed topic names.INVALID_TOPIC_EXCEPTIONfor incompatible named topics andUNKNOWN_TOPIC_OR_PARTITIONfor missing tables or databases.DatabaseNotExistExceptionduring table listing as an empty database, including deletion after database enumeration. Preserve other gateway failures.Increment relative to PR03: 10 files changed, 1896 insertions(+), 4 deletions(-). The shared DDL mapping implementation belongs to #4277; this increment keeps the Metadata protocol, gateway adapter and regression coverage together.
Tests
Validated the source tree committed as
988918e255c502c9b8030b00e68d0d0eed0e21c7on Java 11:mvn -o -pl fluss-kafka -am verify '-Dtest=Kafka*Test,Kafka*ITCase' -Dsurefire.failIfNoSpecifiedTests=false: 90 unit tests and 4 integration tests passed using dependencies from the same reactor.git diff --checkpassed.The full repository test suite and a Flink SQL runtime were not run.
API and Format
Adds Kafka Metadata v0-v11 for tables satisfying the current DDL mapping contract. Existing Fluss RPC and storage formats are reused. Kafka CreateTopics/DeleteTopics, automatic table creation, Metadata v12+, Produce, and JSON record conversion remain outside this PR.
Documentation
The supported versions, qualified topic names, DDL admission rules and per-topic errors are described in this PR, source Javadoc and regression tests. Produce delivery documentation follows with later capability work.
Stack and review
This PR remains a draft until its prerequisites land. It targets Apache
main, so GitHub's Files changed view includes the prerequisites. Metadata is rebased onto PR034361fa11b227f6d5573757b21499f692a289cc12; review it using the incremental comparison against DDL mapping. Update the baseline after prerequisites merge and validate again before marking ready.Generative AI disclosure: Codex assisted with implementation, testing, and commit organization. Human review is required before merge.